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Overview 


ELICIT,  the  Experimental  Platform  for  Investigating  Collaboration,  Information-sharing  and 
Trust  was  originally  developed  as  a  platform  for  conducting  formal  experiments  on  the  effects  of 
using  edge  vs.  traditional  hierarchy  organization  structures.  Over  time  it  became  clear  that 
restricting  the  organizational  modes  to  the  traditional  hierarchy  and  edge  structures  did  not  allow 
for  the  exploration  of  hybrid  structures,  so  the  ELICIT  software  platform  was  enhanced  to 
support  flexible  structures  and  additional  forms  of  communication.  The  post  experiment 
discussions  at  research  universities  evolved  into  the  use  of  ELICIT  as  a  classroom  tool. 

Though  ELICIT  has  been  used  at  both  civilian  and  military  educational  institutions  around  the 
world,  the  most  significant  pedagogical  use  has  been  in  military  schools,  so  most  of  the  examples 
in  this  paper  are  from  military  training.  Based  on  the  author’s  experience  with  civilian 
leadership  training,  it  is  the  author’s  belief  that  the  results  apply  to  both  military  and  civilian 
management  training. 

This  paper  describes  the  evolution  of  ELICIT  as  a  pedagogical  tool  and  provides  guidance  to 
institutions  considering  ELICIT  for  classroom  use.  The  target  audience  is  persons  who  are 
considering  creating  a  course  on  agile  leadership  or  just  using  ELICIT  as  a  one  session 
augmentation  to  an  existing  leadership  program. 

This  paper  is  based  on  my  personal  experiences  using  ELICIT  with  students,  working  with 
researchers  and  educators  to  adopt  ELICIT  for  use  in  their  classrooms,  and  follow-up  interviews. 
The  paper  provides  a  framework  for  viewing  the  evolution  of  ELICIT  as  a  pedagogical  tool, 
what  has  been  done  and  what  works. 


Original  Intent  of  ELICIT  Platform 

ELICIT  was  developed  by  the  United  States  Department  of  Defense  Command  and  Control 
Research  Program  (CCRP)  of  the  Office  of  the  Assistant  Secretary  of  Defense  for  Networks  and 
Information  Integration  (OASD/NII)  as  part  of  their  initiative  to  develop  and  test  principles  of 
organization  that  enable  transformation  from  traditional  hierarchy-based  command  and  control 
practices  to  more  agile  practices  that  transfer  more  power  and  decision  rights  to  the  edge  of  the 
organization.  The  need  for  agility  in  Information  Age  militaries  is  becoming  increasingly 
important.  As  discussed  by  Alberts  and  Hayes  in  Understanding  Command  and  Control,  in  an 
era  of  complex,  coalition,  civil-military  operations,  understanding  how  to  organize  for  agility  not 
just  within  a  specific  organization  but  also  across  differing  organizations  and  cultures  is  a  key 
success  factor. 


There  had  been  a  shortage  of  formal  experimentation  data  on  the  efficacy  of  different  Command 
and  Control  (C2)  organizational  approaches.  In  order  to  remedy  this  shortage,  the  CCRP  created 
and  continues  to  sponsor  and  maintain  the  ELICIT  research  environment.  ELICIT  is  a  Java- 
based  software  platfonn  that  can  be  used  to  run  multi-user  experiments  focused  on  information, 
cognitive,  and  social  domain  phenomena.  People  participate  in  research  sessions  mediated  by 
ELICIT  by  working  together  in  teams  that  can  be  configured  to  reflect  different  organizational 
approaches  (e.g.,  hierarchy,  edge,  hybrid  and  others)  and  that  can  be  subjected  to  a  wide  variety 
of  configurable  conditions. 

The  ELICIT  platform  supports  configurable  scenarios  that  focus  on  the  task  of  identifying  the 
“who”,  “what”,  “where”,  and  “when”  of  a  fictional  adversary  threat.  Information  in  the  fonn  of 
“factoids”  is  provided  periodically  to  each  of  the  participants  during  an  experiment  session.  The 
factoids  and  their  distribution  are  structured  so  that  no  one  participant  receives  all  the 
information  necessary  to  perform  the  task;  thus,  information  sharing  is  required  in  order  for  any 
participant  to  be  able  to  determine  a  solution  to  the  ELICIT  problem. 

ELICIT  provides  an  instrumented  task  environment  that  captures  and  time  stamps  participants’ 
information  sharing  activities.  The  environment  generates  detailed  transaction  logs  summarizing 
this  information.  These,  together  with  participant  surveys  that  can  be  administered  either  prior  to 
a  trial  (for  calibration),  after  a  trial,  or  in  situ,  can  be  used  to  measure  information  sharing, 
collaboration  behaviors  and  situational  awareness,  as  well  as  a  variety  of  value  metrics  including 
the  ability  of  individuals  and  teams  to  correctly  identify  the  future  adversary  attack  and  the  time 
required  to  do  so.  A  log  analyzer  can  be  used  to  quickly  analyze  the  logged  data  so  that  the 
results  are  available  immediately  for  classroom  discussion. 

ELICIT  supports  both  student  participants  and  also  software  agent  participants.  The  software 
agents  can  be  used  in  conjunction  with  students  so  that  the  ELICIT  can  be  played  regardless  of 
the  number  of  students  registered  for  a  particular  course  (with  the  agents  filling  in  for  any 
missing  students  needed  to  complete  a  particular  organization  structure.)  These  software  agents 
are  semi-intelligent  agents  with  a  sensemaking  capability.  Based  on  data  they  receive,  the  agents 
construct  a  “mental  model”  of  the  threat  scenario  and  thus  use  “situational  awareness”  to 
determine  their  actions.  The  agents  have  ~50  configurable  variables  to  define  their  personalities 
and  styles  of  social  interaction  with  the  other  experiment  participants.  Using  these  variables, 
agents  can  be  configured  to  operate  in  human  timeframes  (e.g.,  seconds  and  minutes),  rather  than 
just  computer  timeframes  (e.g.,  nanoseconds  and  microseconds),  show  human  levels  of 
variability  and  human  personality  traits  such  as  tendencies  to  hoard  infonnation,  reciprocate 
favors  and  trust  team  members  (among  others). 

Considerable  research  has  been  conducted  to  date  using  ELICIT.  A  selection  is  included  in  the 
bibliography.  The  ELICIT  platform  continues  to  evolve  to  meet  the  ongoing  needs  of  researchers 
and  educators.  Though  ELICIT  was  originally  conceived  as  a  research  tool,  often  the  exercise 
debriefings  evolved  into  energetic  classroom  discussions,  particularly  when  a  suitably 


experienced  leader  was  present  to  lead  the  conversation.  Since  most  of  the  ELICIT  research 
relies  on  military  and  business  students  as  participants,  overtime  it  was  realized  that  ELICIT 
could  also  be  leveraged  for  use  as  a  pedagogical  tool  in  the  classroom. 


Role  of  Computer  Simulations  in  the  Classroom 

There  is  a  long  history  of  using  computer  games  and  simulations  for  training  and  as  part  of 
formal  degree  programs.  Often  the  dynamics  of  real-time  complex  systems  such  as  distributed 
team  management  are  counter-intuitive.  Simulations  in  which  students  personally  experience  the 
dynamics  of  successive  rounds  of  interaction  can  be  very  effective  at  teaching  lessons  in  network 
operations.  Simulations  and  games  can  also  provide  a  useful  counterpoint  to  “death  by 
PowerPoint.”  Thus  business  schools  have  for  some  time  used  simulation  games  to  augment 
classroom  lectures.  For  example,  the  School  of  Management  at  Massachusetts  Institute  of 
Technology  (MIT)  has  long  used  the  Beer  (distribution)  Game  to  teach  a  number  of  key 
principles  of  supply  chain  management.  Participating  in  an  exercise  in  which  your  leadership 
decisions  impact  your  game  score,  under  circumstances  where  you  can  compare  your 
performance  to  other  members  of  your  class,  creates  a  strong  visceral  experience  for  student 
leaders.  Students  better  understand  the  dynamics  and  implications  of  their  choices  because  they 
have  personally  experienced  them. 


ELICIT  Use  in  the  Classroom 

The  ELICIT  platform  is  flexible  enough  that  it  can  be  used  in  a  variety  of  educational  formats.  It 

has  been  used  as  part  of  an  afternoon  session,  as  part  of  a  one  day  course  and  integrated  into  a 

semester  course  as  part  of  a  multiple  class  experience.  The  original  baseline  ELICIT  experiment 
protocol  consisted  of  the  following: 

•  Introduction 

•  Pre-briefing  video 

•  Opportunity  to  read  written  instructions 

•  Practice  round  (20  minutes) 

•  Break 

•  Actual  round  (60  minutes) 

•  Post  experiment  survey 

•  Post-briefing  video 

•  Optional  discussion  session 


The  first  eight  steps  typically  take  two  to  two  and  a  half  hours  to  conduct.  For  use  in  the 
classroom,  this  eight  step  sequence  was  modified  to  fit  into  a  typical  one  hour  class  period  by 
removing  the  break  and  the  practice  round;  and  shortening  the  introduction,  survey  and  actual 
round  to  40  minutes  (note  there  was  generally  not  much  activity  in  the  last  twenty  minutes  of  a 
60  minute  round  anyway.)  It  was  found  that  making  these  changes  did  not  change  the 
fundamental  results  and  made  it  much  easier  to  schedule  students  as  participants.  For  courses 
with  longer  classroom  sessions,  these  modifications  left  more  time  for  preparatory  lecture  and 
meaningful  discussion. 

The  first  few  times  ELICIT  was  used  at  a  university  (Boston  University),  the  post  experiment 
debriefing  (with  civilian  business  students)  was  extended  into  a  discussion  period  so  that  the 
experiment  designers  and  ELICIT  platform  developer  could  get  general  feedback.  These 
discussions  were  lively  and  spirited  and  it  was  observed  that  there  was  potential  for  using 
ELICIT  as  a  training  tool,  not  just  a  research  tool. 

The  first  time  that  the  post  experiment  discussion  session  morphed  into  a  classroom  experience 
in  its  own  right  (rather  than  merely  a  debriefing  on  the  experiment  itself  and  how  the  experiment 
could  be  improved)  was  when  the  baseline  ELICIT  experiment  was  run  at  West  Point.  The 
experiment  was  run  with  a  group  of  first  year  cadets  (half  in  an  edge  organization  and  half  in  a 
traditional  hierarchy.)  When  the  exercise  was  completed,  a  Colonel  led  an  energetic  discussion 
on  the  subject  of  intelligence  gathering  and  how  one  detennines  if  one  has  identified  a  threat 
with  sufficient  certainty  that  it  is  appropriate  to  recommend  its  destruction.  He  took  an  exercise 
in  how  organizational  structure  impacts  task  effectiveness,  and  explored  not  only  the  pros  and 
cons  of  organizational  structure,  but  discussed  the  effect  of  training  on  structure  effectiveness, 
what  types  of  behaviors  they  considered  constructive  or  destructive  in  team  members  and  how 
one  makes  decisions. 

Part  of  what  made  this  discussion  interesting,  was  that  the  cadets  had  just  had  the  personal 
experience  of  trying  to  identify  the  advisory  attack  in  an  ELICIT  scenario  under  time  pressure, 
with  limited  information  and  the  knowledge  that  their  competitive  classmates  were  also 
performing  the  same  task  (note  student  participants  are  anonymized  to  protect  their  privacy.) 
Cadets  were  able  to  discuss  their  own  experience  of  working  in  the  traditional  hierarchy  or 
distributed  edge  organization  and  explore  their  theories  for  why  one  worked  better  than  the  other 
in  this  particular  scenario.  They  were  also  able  to  compare  and  contrast  what  it  felt  like  to 
operate  under  the  two  organization  structures. 

It  was  from  this  session,  that  the  ELICIT  Community  of  Interest  (COI)  first  began  to  believe  that 
in  addition  to  using  ELICIT  as  a  platform  for  conducting  formal  research,  ELICIT  could  also  be 
valuable  as  a  formal  instructional  aid. 

The  first  time  that  ELICIT  was  used  to  facilitate  a  training  discussion  was  at  the  Portuguese 
Military  Academy  in  July  2006.  At  that  time,  ELICIT  required  the  installation  of  client  software 


and  the  automated  capabilities  for  analyzing  the  log  files  were  in  a  development  stage,  so 
preparing  a  lab  for  an  ELICIT  exercise  was  fairly  labor  intensive  and  only  a  very  basic  analysis 
of  participant  performance  was  available  immediately  after  an  exercise.  This  limited  the  scope  of 
post  exercise  discussion.  An  ELICIT  expert  was  brought  in  to  facilitate  the  discussions. 

The  first  time  the  hosted  version  of  ELICIT  was  used  a  part  of  a  formal  class  was  in  a  one  day 
session  on  distributed  leadership  for  mid  career  officers  with  high  potential  held  at  the  Naval 
War  College  in  January  2008. 

This  class  consisted  of  about  32  students.  After  a  discussion  on  leadership  and  organization,  the 
class  was  divided  into  two  groups:  one  operating  in  a  traditional  hierarchy  and  the  other  in  an 
edge  organization.  Since  the  classic  ELICIT  scenarios  require  17  participants  each  and  the  class 
size  was  a  little  less  than  34,  the  two  groups  were  augmented  by  a  couple  of  software  agents  so 
that  there  were  17  participants  (mostly  human)  in  each  group.  The  software  agents  were 
configured  to  be  supportive  participants  that  passed  on  all  the  information  they  received.  In 
summary,  the  organization  of  the  one  day  session  was  a  follows: 

•  Discussion 

•  ELICIT  trial  (half  the  students  participating  in  edge  and  half  in  a  traditional  hierarchy 
organizations) 

•  Lunch  break  (during  which  the  log  files  for  the  two  groups  were  analyzed  and  turned  into 
a  power  point  presentation  that  compared  and  contrasted  the  performance  of  the  two 
groups  in  the  classroom  and  the  behavior  of  pervious  research  participants. 

•  Presentation  of  the  results  followed  by  a  student  and  leader  discussion. 


Examples  of  the  discussion  slides  are  available  from  the  ELICIT  COI.  Some  of  the  elements  of 
the  day  that  made  it  effective  and  practical  included  the  following: 

•  The  software  was  available  at  no  charge  from  the  CCRP  so  that  procurement  issues  were 
not  an  impediment 

•  The  ELICIT  software  was  available  over  the  Internet  so  the  school  only  need  to  supply 
internet  connections 

•  The  class  was  large  enough  to  support  running  ELICIT  with  two  groups  that  had  different 
organizational  structures  so  there  were  a  variety  of  experiences  in  the  classroom  on  which 
to  base  discussion 

•  The  class  was  big  enough  to  include  a  variety  of  perspectives,  and  small  enough  so  that 
most  could  participate  in  the  discussion 

•  Discussion  leaders  in  the  room  included  senior  naval  officers  and  a  retired  admiral  as 
well  and  Dr.  Richard  Hayes  and  Dr.  David  Alberts,  authors  of  Power  to  the  Edge  so  that 
there  was  lively,  infonned  discussion  backed  by  real  world  leadership  expertise 

•  A  detailed  analysis  of  the  performance  of  the  two  groups  was  available  for  the  discussion 
so  that  questions  could  be  answered  and  theories  on  causes  of  performance  variations 
addressed 


•  Results  for  previous  research  runs  were  also  made  available  to  address  competitive 
curiosity  and  show  that  the  results  they  experienced  where  not  happenstance  but  also 
reproducible  experimentally. 


The  most  integrated  used  of  ELICIT  in  a  course  to  date  is  in  the  Netcentric  Warfare  and 
Operations  (NCW)  elective  course,  at  National  Defense  University  (NDU).  The  course, 
developed  by  Dr.  Cathryn  Downes,  Professor,  Information  Strategies  Department,  “examines  the 
tenets  and  technologies  of  network  centric  warfare  and  operations  (NCW)  as  an  evolving  way  of 
warfare  and  a  central  component  of  Department  of  Defense  (DoD)  innovation  for  future 
operations.  The  nature  and  dynamics  of  information-enabled  networks  are  examined,  as  are 
other  emergent  technologies  that  are  influencing  how  NCW  evolves.”  It  is  a  full  24  hour  credit 
course  consisting  of  12  two  hour  sessions. 

This  course  has  been  offered  twice  as  of  the  writing  of  this  paper,  and  its  use  of  ELICIT  has 
evolved  both  in  terms  of  the  ELICIT  software  used  and  the  software’s  use  in  the  course.  The 
course’s  students  are  typically  mid  career  officers  from  the  United  States  and  coalition  partners. 
In  its  most  recent  offering,  the  students  were  mostly  05  and  06  level  chiefs  of  staff.  ELICIT  is 
used  to  provide  a  “mini  command  problem”  that  augments  the  logic  of  the  classroom  lesson  with 
real  experience.  Its  use  in  the  course  is  timed  to  provide  an  “ah  ha”  to  the  class.  One  student, 
after  taking  the  course  reported  that  he  spent  a  whole  year  in  Iraq  doing  nothing  but  “feeding”  his 
commander,  and  that  he  now  understood  that  he  should  have  been  finding  out  what  everyone  else 
needed.  One  student  had  previously  thought  that  he  would  never  be  promoted  past  the  Brigadier 
General  level,  and  now  sees  how  he  could  command  at  even  higher  levels.  Professor  Downes 
reports  that  “ELICIT  prompted  them  [the  students]  to  think  about  the  costs  and  consequences  of 
hierarchy  vs.  networked  organizations.”  NDU’s  Netcentric  Warfare  and  Operations  (NCW)  class 
won  an  award  the  first  time  it  was  run. 

NDU  Methodology 

In  its  current  formulation,  ELICIT  is  used  three  times  during  the  course  in  successive  classroom 
sessions.  In  the  first  session,  ELICIT  is  run  in  a  hierarchy  configuration  to  show  the  students  a 
baseline.  The  second  use  is  with  an  edge  (or  distributed)  organization.  This  provides  the  students 
a  sharp  contrast  to  the  experience  with  the  traditional  hierarchy  configuration.  After  the  second 
usage,  and  discussion  of  the  strengths,  weakness  and  issues  of  organization  structure,  students 
are  invited  to  spend  ten  minutes  suggesting  what  they  consider  to  be  an  ideal  hybrid 
organization.  Then  before  the  next  class,  the  new  organization  structure  is  configured,  and  the 
students  have  the  opportunity  to  see  how  their  proposed  hybrid  structure  works  and  experience 
what  it  is  like  to  operate  under  it.  The  students  are  then  led  in  a  classroom  discussion  of  how  the 
new  org  structure  met  or  did  not  meet  their  expectations. 


In  the  most  recent  class,  the  hybrid  design  was  implemented  using  the  ELICIT  organization 
configuration  tile  and  the  use  of  a  new  feature  called  progress  check.  Progress  check  allows 
participants  to  share  their  current  theory  about  the  adversary  attack  with  other  participants  and 
for  participants  to  see  the  other  player’s  latest  theories.  This  is  somewhat  similar  to  the  post  card 
approach  used  by  Leweling  in  her  ELICIT  research  in  which  participants  were  able  to  share  a 
post  card  containing  their  best  theory  about  the  adversary  attack  with  another  participant,  (i.e. 
very  controlled  disclosure  of  theories.)  For  their  hybrid  design,  the  students  also  chose  to  use  the 
structured  format  for  their  Identities  (identification  of  the  adversary)  and  to  enable  Tweet  sized 
free  form  text  messages.  The  ELICIT  features  they  chose  are  relatively  recent  enhancements. 
ELICIT  now  supports  a  much  wider  variety  of  communications  forms  than  used  in  the  original 
experiment. 

Note  that  since  it  was  a  multi-national  class,  some  participants  had  better  English  skills  than 
others  so  it  was  necessary  to  ensure  that  the  person  in  the  Cross-team  Coordinator  role  in  the 
hierarchy  had  good  English  skills. 

The  most  effective  use  of  ELICIT  is  as  a  counterpoint  to  pre-conceptions.  Thus  running  ELICIT 
in  two  organizational  modes,  so  the  students  can  compare  and  contrast  them,  is  a  core  part  of 
using  ELICIT  in  the  classroom.  This  effect  is  so  dramatic  that  the  third  round  may  not  be 
necessary.  On  the  other  hand,  now  that  ELICIT  offers  a  much  richer  set  of  organizational 
structure  options  and  communications  methods,  it  is  possible  to  use  ELICIT  to  practice  designing 
task  specific  communications  networks  and  evaluating  their  efficiency.  Thus,  NDU  is 
considering  using  ELICIT  for  a  fourth  round  so  that  students  can  further  practice  this  new  skill. 


Key  Success  Factors 

•  Structuring  the  class  to  provide  experiences  in  both  edge  and  traditional  command 
and  control  hierarchy  organizational  structures.  The  core  use  is  to  provide  the 
students  the  opportunity  to  experience  both  edge  and  hierarchy  organization  structures  so 
that  students  can  compare  and  contrast  structures  for  task  effectiveness.  This  means 
structuring  the  course  so  that  either  multiple  teams  use  the  different  structures 
simultaneously  or  that  each  student  has  the  opportunity  to  sequentially  experience 
operating  under  varying  organizational  structures. 

•  Integrating  ELICIT  into  the  course.  Giving  careful  consideration  to  how  ELICIT  fits 
into  your  over  all  curriculum  and  educational  objectives.  What  is  the  ideal  place(s)  in  the 
course  to  use  ELICIT  to  augment  the  education  experience?  What  is  the  motivation  for  its 
use?  Which  of  the  many  lessons  do  you  wish  to  highlight? 


•  Having  a  strong,  credible  discussion  leader.  For  military  students,  the  credible 
discussion  leader  is  a  dynamic  leader  with  recent  combat  experience  (i.e.  “street  cred”.) 
In  addition,  for  a  class  with  military  personal,  the  discussion  leader  needs  to  have 
sufficient  rank  when  compared  to  the  students.  For  the  NDU  course  that  means  a  Colonel 
or  above  with  recent  deployment  in  a  war  zone.  (NDU  was  fortunate  enough  have  the 
honor  of  having  a  General  join  them  for  one  class.)  Even  for  cadet  level  participants  at 
the  Military  Academy,  a  Colonel  can  be  a  very  effective  addition.  It  is  possible  that 
Colonel  is  the  minimum  rank  for  leading  this  type  of  discussion  at  a  military  academy. 
The  Naval  War  College  session  was  augmented  with  a  retired  Admiral.  It  would  be 
interesting  to  get  more  data  points  on  this  issue.  The  higher  the  rank  of  the  discussion 
leader,  the  more  the  ELICIT  exercise  can  be  used  as  a  jumping  off  point  for  a  high  level 
general  leadership  discussion. 

•  Arranging  for  technical  support.  This  includes  someone  to  configure  the  class 
exercise(s)  in  ELICIT  and  to  analyze  and  interpret  the  log  files  during  the  class.  It  is 
important  for  the  learning  experience  that  the  students  see  the  results  of  their  performance 
under  the  various  organizational  structures  while  the  experience  of  participating  in  the 
process  is  still  fresh. 

•  Arranging  to  use  scenarios  that  are  relevant  and  realistic  for  the  class.  All  classroom 
scenarios  are  by  definition  abstracted  from  reality.  It  is  important  that  the  selected 
scenario  be  one  to  which  the  students  relate.  The  ELICIT  intelligence  task  is  well  suited 
to  a  variety  of  military  and  civilian  leadership  students;  however  the  core  scenarios  won’t 
be  suitable  for  all  students.  The  existing  scenarios  have  only  true  information,  so  the 
scenarios  don’t  yet  support  discussion  of  counter  intelligence  tactics. 

•  Arranging  for  appropriate  class  sizes.  Though  agents  can  be  used  to  substitute  for 
human  participants  in  the  classic  ELICIT  hierarchy  scenario,  due  to  the  nature  of  the 
organization  structure,  nine  is  really  the  minimum  number  of  students  that  makes  sense, 
and  it  is  better  to  have  at  least  13. 


Lessons  Illustrated  by  ELICIT 

There  are  a  number  of  lessons  that  can  be  illustrated  with  ELICIT.  As  with  all  teaching,  the  most 
effective  training  starts  with  the  experience  base  of  the  class,  is  adapted  to  the  education 
objectives,  and  takes  into  account  the  range  of  experience  in  the  group.  Because  ELICIT  is  a 
configurable  platform,  it  can  be  used  for  management  and  leadership  training  at  a  variety  of 
levels.  To  date,  ELICIT  training  has  been  used  with  college  freshman  all  the  way  to  mid  career 
military  officers.  Participants  have  included  PHD  students,  professors  and  Special  Forces.  (Using 
professors  as  participants  in  ELICIT  is  not  recommended.  Yes,  we  learned  this  the  hard  way.)  It 


has  been  used  with  persons  who  were  trained  to  use  traditional  hierarchy  organizations  as  the 
first  choice  for  organizational  structure  as  well  as  persons  who  had  no  formal  training  in 
operating  as  part  of  a  hierarchy  and  are  accustomed  to  operating  in  distributed  organizations.  It 
has  been  used  with  civilian  and  military  academies  in  the  US  and  abroad.  See  Appendix  A  for  a 
partial  list  of  participating  schools. 

The  core  ELICIT  lesson  is  that  some  tasks  are  more  efficiently  and  effectively  performed  by 
teams  with  a  distributed  organizational  structure  rather  than  a  traditional  hierarchy.  Since 
ELICIT  has  been  used  to  conduct  formal  experiments,  it  has  been  used  to  explore  other 
conditions  that  impact  organizational  agility  and  there  are  additional  auxiliary  lessons.  These 
include  that: 

•  The  organizational  structure  (network  structure)  you  use  matters.  You  shouldn’t  assume 
that  the  organizational  structure  you  are  given  (or  start  with)  is  the  right  one  for  every 
task  (Cranfield  University.) 

•  The  power  of  distributed  leadership  is  valuable  especially  in  solving  complex  and 
network  problems. 

•  More  bandwidth  doesn’t  always  make  it  easier  to  do  the  task.  This  is  frequently  asserted, 
but  the  addition  of  unrestricted  chat  communications  (Singapore  Armed  Forces  Centre 
for  Military  Experimentation)  has  been  shown  to  reduce  the  effectiveness  of  ELICIT 
teams.  (Note  that  historically  the  military  has  used  various  practices  to  manage 
communications  on  various  official  channels  to  reduce  the  possibility  of  communications 
overload.) 

•  Situation  conditions  and  training  impact  the  optimal  choice  of  organization  structure 
(NPS  and  Portuguese  Military  Academy.) 

•  Many,  though  not  all,  of  the  tasks  perfonned  by  today’s  military,  military  analyst  or 
knowledge  worker  are  better  perfonned  in  edge  rather  than  traditional  command  and 
control  hierarchy  organizational  structures. 

•  The  starting  level  of  group  trust  influences  the  effectiveness  of  an  organizational  structure 
(NPS.) 

ELICIT  has  been  used  at  wide  range  of  experience  levels.  One  can  hold  leadership  conversations 
with  students  at  multiple  levels  depending  on  student  and  instructor  experience.  The  results 
produced  by  ELICIT  are  dramatic  and  reproducible  in  the  classroom.  As  ELICIT  research 
progresses  and  evolves,  there  will  be  additional  lessons  from  ELICIT  research  that  can  be 
brought  into  the  classroom. 


Example  Discussion  Leader  Topics 

•  How  will  more  connectivity  impact  task  performance?  Is  the  answer  different  for 
timeliness  and  accuracy? 

•  In  what  situations  are  edge  organizations  more  appropriate  and  in  which  situations  are 
traditional  command  and  control  hierarchies  more  effective? 


•  How  does  previous  training  influence  ability  to  perfonn  under  different  organizational 
structures? 

•  Should  team  trust  influence  the  organization  structure  chosen?  If  so  how? 

•  How  well  do  you  trust  the  answers  you  determined  during  the  exercise? 

•  Did  anyone  in  your  group  hoard  information?  If  so,  why?  What  was  the  effect?  (The 
phenomenon  of  “hoarding”  information  occurs  in  many  ELICIT  exercises  and  its 
discussion  reminds  students  that  in  the  real  world,  directives  don’t  always  lead  to  the 
results  expected.) 

•  Did  anyone  in  your  group  spam  the  group?  If  so  why?  What  was  the  effect? 

•  What  types  of  team  member  behavior  are  not  effective  in  particular  organization 
structures? 

•  How  should  an  ideal  team  member  behave  in  particular  organizational  structures? 


Next  Steps 

ELICIT  has  evolved  significantly  as  a  research  and  a  pedagogical  tool  and  continues  to  mature. 
One  possible  further  extension  is  to  support  personalized  training.  Each  student  could  access 
their  own  virtual  version  of  ELICIT  with  16  software  agents  configured  to  be  supportive  or  not 
supportive  of  the  mission,  etc.  in  accordance  with  the  goals  of  the  training.  Then  each  student 
could  experience  exactly  the  same  conditions  as  all  the  other  students.  While  it  is  possible  to 
infleunce  student  behavior  by  varying  the  instructions  to  the  students,  adult  students  don’t  always 
follow  instructions,  so  this  would  provide  much  more  controlled  learning  experiences.  This 
approach  would  also  support  training  of  more  variable  sized  groups,  and  a  wider  variety  of 
experiences. 

Additional  suggestions  that  have  been  made  by  classroom  leaders  include: 

•  Making  the  participant  user  interface  more  aesthetic.  (The  original  interface  is  sparse  and 
functional  to  eliminate  unintended  variables.  It  could  be  made  more  appealing). 

•  Adding  additional  and  more  realistic  scenarios,  especially  those  that  include  conflicting 
infonnation. 

•  Enhancing  the  ELICIT  agents  so  that  they  are  also  able  to  perform  the  newer 
communications  functions. 

•  Continuing  the  interaction  of  ELICIT  researchers  and  ELICIT  educators  to  bring 
additional  ELICIT  lessons  to  the  classroom. 

•  Using  ELICIT  to  teach  students  the  new  skill  of  how  to  optimize  the  organizational 
approach  for  the  task  and  situation. 


One  of  the  purposes  of  ELICIT  is  to  perform  formal  research  in  the  area  of  agility  and  to 
advocate  for  the  appropriate  use  of  varying  organization  approaches.  Therefore  CCRP  has  used  a 
distributed  approach  to  organizing  the  ELICIT  Community  of  Interest  and  has  been  very 
supportive  of  innovative  uses  of  ELICIT.  Suggestions  for  further  evolution  of  ELICIT  and  its 
uses  are  strongly  encouraged. 


Conclusions  &  Discussion 

Though  military  leaders  have  been  writing  about  the  art  of  leadership  since  Sun  Tsu,  network 
science  and  agile  leadership  are  relatively  new  fields  of  study.  Often  students,  even  or  especially 
ones  with  years  of  command  experience,  come  to  class  with  preconceived  notions  about  how 
best  to  organize  their  teams.  Use  of  ELICIT-based  leadership  exercises  in  the  classroom  can  be  a 
useful  way  to  help  students  viscerally  internalize  lessons  about  the  effects  of  organization 
structure  on  team  effectiveness.  ELICIT  helps  students  appreciate  the  effects  of  varying 
organizational  structure. 

ELICIT  has  evolved  from  being  just  a  research  tool  to  also  being  a  pedagogical  tool.  ELICIT 
was  conceived  as  a  platform  that  would  increase  research  leverage  because  it  could  be  reused  by 
multiple  researchers.  Since  the  same  tool  has  been  optimized  for  use  in  both  in  the  research  lab 
and  the  classroom,  ELICIT  can  be  used  both  as  a  research  tool  and  as  a  tool  for  bringing  these 
new  research  results  to  life  in  the  classroom  in  a  seamless  fashion.  The  same  configurable 
features  that  allow  ELICIT  to  be  used  to  conduct  a  wide  variety  of  experiments  can  also  be  used 
to  enable  students  to  explore  approaches  they  believe  could  improve  their  leadership 
effectiveness. 
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Appendix  A  -  Educational  Institutions  that  have  Used  ELICIT 


The  CCRP  is  grateful  for  the  participation  of  the  following  institutions  and  to  those  individuals 
who  provided  contact  information  for  this  paper: 

Civilian 

•  Boston  University 

•  Harvard  University 

•  Loyalist  College,  Canada 

•  University  of  South  Hampton,  UK 

•  York  University,  Canada,  contact  Chris  Fredette,  chris  fredette  Tati  carlton.ca 

Military 

•  Army  Research  Labs  (ARL),  contact  Kevin  Chan,  kevin.s.chan  Tati  us.army.mil 

•  Cran Held  University 

•  Defense  Academy  of  the  United  Kingdom 

•  George  Mason  University 

•  Johns  Hopkins  University 

•  Military  Polytechnic  Academy,  Army  of  Chile 

•  National  Defense  University,  contact  Cathryn  Downes,  downesc  Tati  ndu.edu 

•  Naval  Post  Graduate  School,  contact  Mark  Nissen,  mnissen  Tati 
nps.edu,  http://www.nps.edu/Academics/Centers/CEP/ 

•  Naval  War  College 

•  Portuguese  Military  Academy,  contact  Marco  Manso,  marco.manso  Tati  weare- 
company.com 

•  Singapore  Anned  Forces  Centre  For  Military  Experimentation,  contact  Mervyn 
Cheah,  mervyncheah  Tati  yahoo.com.sg 

•  US  Army  War  College,  contact  Cathryn  Downes,  downesc  Tati  ndu.edu 

•  US  Military  Academy 

•  Defense  Research  and  Development  Canada,  Toronto,  contact  Matthew 
Duncan,  matthew  duncan  Tati  drdc-rrdc.gc.ca 


Appendix  B  -  Running  ELICIT  in  a  Classroom  Environment 


This  appendix  provides  pointers  on  some  practical  details  for  using  ELICIT  in  the  classroom  as  a 
pedagogical  tool. 

Your  Infrastructure 

The  minimum  requirement  is  a  classroom  with  internet  access  for  each  participant.  Labs  where  a 
computer  is  provided  for  each  student  are  ideal,  but  most  students  in  most  settings  can  bring  a 
laptop  if  necessary.  If  Wi-Fi  connections  are  used,  care  should  be  taken  to  ensure  that  bandwidth 
is  sufficient.  In  some  classrooms,  additional  routers  have  been  necessary.  There  should  also  be 
a  monitor  or  projector  for  displaying  video  and  slides. 

ELICIT  Software 

The  ELICIT  software  and  the  survey  software  are  available  as  hosted  services  on  the  Internet. 
There  is  no  charge  to  members  of  the  ELICIT  COI  for  using  the  Software.  Information  about 
joining  the  COI  is  available  at  http://www.dodccrp.org/html4/elicit.html.  The  primary 
requirement  for  joining  the  COI  is  to  provide  copies  of  the  raw  data,  analyses,  learnings,  etc.  to 
the  COI  so  that  the  community  can  leverage  the  experiences  of  its  community. 

For  locations  that  block  internet  access,  it  is  possible  to  install  ELICIT  and  the  survey  server  on  a 
local  intranet.  ELICIT  supports  and  has  been  tested  with  the  Internet  Explorer  and  Firefox 
browsers. 

The  ELICIT  software  comes  with  documentation  and  supporting  materials  for  use  in  formal 
experiments  and  classroom  exercises.  These  materials  include: 

•  Pre  and  post  briefing  videos 

•  Welcome  and  order  of  proceedings  slides 

•  Checklists 

•  Standard  scenario  configurations 

•  Log  analyzer 

•  Sample  analysis  briefing 

The  ELICT  platform  and  its  supporting  materials  should  be  considered  a  tool  kit  to  add  richness  to  a 
course  rather  than  a  course  in  and  of  itself.  Contact  the  author  for  suggestions  on  adopting  ELICIT  to 
your  specific  circumstances. 

Participants 

The  classic  ELICIT  hierarchy  has  17  roles.  So  classroom  exercises  that  include  the  classic 
scenarios  require  multiples  of  17  students.  As  discussed  it  is  possible  to  substitute  software 


agents  for  some  of  the  students.  Groups  have  been  run  with  as  few  as  9  human  participants  and  8 
agents. 

It  is  technically  possible  to  run  ELICIT  with  1  human  and  hundreds  of  agents),  but  we  have  not 
yet  been  using  ELICIT  this  way  in  a  classroom  setting.  In  the  future  it  would  be  possible  to 
provide  individual  students  fully  controlled  ELICIT  experiences  by  having  all  of  the  other 
participants  be  agents  that  have  been  configured  with  specific  personalities. 

Timing 

The  Boston  University  experiment  runs  (first  school  runs)  were  conducted  with  20-minute 
practice  rounds  and  a  60-minute  experiment  round.  A  full  hour  is  too  long  for  most  subjects.  We 
recommend  reducing  the  experiment  round  to  40  minutes.  Subjects  can  became  noticeably 
restless  starting  about  30  minutes  into  the  experiment  runs. 

Twenty  minutes  is  enough  time  for  everyone  to  complete  the  classic  survey. 

Boredom 

With  the  advent  of  smart  phones  and  iPads,  etc.  it  is  regrettably  helpful  to  instruct  participants  to 
not  use  other  devices  during  an  exercise. 

Frequently  Asked  Questions 

By  software  version  1.13  the  user  interface  was  so  improved  that  questions  about  functionality 
virtually  stopped.  The  moderator/coordinator  checklist  has  been  successively  annotated  to 
anticipate  most  of  the  other  questions.  The  result  is  that  we  now  have  very  few  questions  during 
the  session  after  the  instructions  have  been  presented. 

Staffing 

With  current  software  and  processes,  two  people  are  sufficient  to  run  a  live  ELICIT  exercise. 
(Instructor  and  ELICIT  operator.)  You  may  want  additional  persons  in  the  room  to  achieve 
specific  discussion  objectives. 

The  most  successful  ELICIT  military  classroom  discussions  have  included  senior  officers 
(Colonel  or  above)  as  guests.  A  senior  officer  with  command  and  field  experience  brings  the 
needed  credibility  to  discussions  of  military  decision  making. 

Areas  for  Improvement 

As  with  all  exercises,  especially  those  evolved  from  controlled  experiments,  there  is  always  a 
dynamic  tension  between  controlling  for  outside  influences  (variables)  and  realism  (reality  is 
messy.)  The  ELICIT  scenarios  currently  used  in  the  classroom  don’t  include  any  conflicting 
information.  This  is  an  area  we  hope  to  address  in  the  future. 


Pedagogical  use  of  ELICIT 


ICCRTS  16 
22  June  2011 


Mary  Ruddy 

Sponsored  by  Evidence  Based  Research 


Agenda 


•  Overview  and  original  intent  of  ELICIT 

•  Evolution  of  classroom  use 

•  Key  success  factors 

•  Lessons  illustrated 

•  Sample  discussion  topics 

•  Where  used 

•  How  to  get  involved 


ELICIT  Introduction 


•  ELICIT-Experimental  Laboratory  for  Investigating 
Collaboration,  Information-sharing  and  Trust 

•  Part  of  the  Command  and  Control  Research  Program 
(CCRP)’s  network-centric  warfare  initiative 

•  Engaged  in  developing  and  testing  principles  of  organization 
that  transfer  power  and  decision  rights  to  the  edge  of  the 
organization 

•  Needed  to  frame  testable  hypotheses  about  the  relative 
effectiveness  of  edge  organizations  in  comparison  to  other 
methods  of  organization  through  a  series  of  real-world 
simulations 

•  Facility  was  created  to  run  these  experiments 
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Base  Experiment 


Hypothesis  testing  experiment 

Live  subjects,  multi-user 

Run  in  edge  or  traditional  hierarchy  mode 

Experiment  software  records  all  actions  for  analysis 


Experiment  Task 


Identify  the  who,  what,  where  and  when  of  an 
adversary  attack 

Task  scenarios  are  anonymized 

Participants  are  anonymized 

Participants  periodically  receive  factoids  about  the 
situation 

No  one  person  has  all  the  information  needed 
Task  success  requires  communication 


ELICIT  Evolution 


•  More  flexibility  in  organization  structure 

•  More  flexibility  in  communications  options 

•  More  flexibility  in  task  scenario 

•  Humans  and  software  agents 

•  Use  as  a  training  tool 


Evolution  of  Classroom  Use 


BOSTON 

UNIVERSITY 


Research  feedback 

•  Ad  hoc 

•  No  leader  or  agenda 

•  Not  part  of  formal  course 

•  Single  point  of  experience 

•  Undergraduates 
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Evolution  of  Classroom  Use 


Opportunity  for  lesson 

•  Part  of  cadet  education, 
but  not  formal  class 

•  Discussion  leader 

•  Multiple  points  of 
experience 


8 


Evolution  of  Classroom  Use 


academia 

militar 


Classroom  session 

•  Guest  discussion  leaders 

•  Extended  planned 
discussion 

•  Multiple  points  of 
experience 


Evolution  of  Classroom  Use 


us.  naval  war  College 

-  £rt.  1884  - 

NEWPORT,  RHODE  ISLAND 


•  Day  long  special  topic 

•  Multiple  guest  discussion 
leaders 

•  Pre  and  post  discussions 

•  Multiple  points  of 
experience 

•  Detailed  analysis  feedback 
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Evolution  of  Classroom  Use 


NATIONAL  DEFENSE 

JUNI  VERSITY 

*  4  J 

*»!■/.*  St'"' 


Part  of  multi-session  course 

•  Integrated  with  lesson  plan 

•  Guest  discussion  leader 

•  Multiple  points  of 
experience 

•  Detailed  analysis  feedback 

•  Multiple  iterations  for 
deeper  insights 

•  Award  winning  course 


11 


Key  Success  Factors 


•  Live  comparison  -  structuring  the  class  to  provide 
experiences  in  both  edge  and  traditional  command 
and  control  hierarchy  organizational  structures 
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Key  Success  Factors  (cont.) 


Integrating  ELICIT  into  the  educational  program 

Having  a  strong,  credible  discussion  leader 

Arranging  for  technical  support 

Arranging  to  use  scenarios  that  are  relevant  and 
realistic  for  the  class. 

Arranging  for  appropriate  class  size 


Lessons  Demonstrated 


•  The  organizational  structure  (network  structure)  you 
use  matters 

-  Don’t  assume  the  organization,  structure  you 
start  with  is  the  right  one  for  every  task 
(Cranfield  University) 

•  The  power  of  distributed  leadership  is  valuable 
especially  in  solving  complex  and  network  problems 

•  The  starting  level  of  group  trust  influences  the 
effectiveness  of  an  organizational  structure  (NPS) 
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Lessons  Demonstrated  (cont.) 


More  bandwidth  doesn’t  always  make  it  easier  to 
do  the  task.  (Singapore  Armed  Forces  Centre  for 
Military  Experimentation  ) 

Situation  conditions  and  training  impact  the  optimal 
choice  of  organization  structure  (NPS  and 
Portuguese  Military  Academy) 

Many,  though  not  all,  of  the  tasks  performed  by 
today’s  military,  military  analyst  or  knowledge 
worker  are  better  performed  in  edge  rather  than 
traditional  command  and  control  hierarchy 
organizational  structures 


Sample  Discussion  Topics  Include 


•  How  will  more  connectivity  impact  task 
performance?  Is  the  answer  different  for  timeliness 
and  accuracy? 

•  In  what  situations  are  edge  organizations  more 
appropriate  and  in  which  situations  are  traditional 
command  and  control  hierarchies  more  effective? 

•  How  does  previous  training  influence  ability  to 
perform  under  different  organizational  structures? 

•  Should  team  trust  influence  the  organization 
structure  chosen?  If  so  how? 
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Sample  Discussion  Topics  (cont.) 


•  How  well  do  you  trust  the  answers  you  determined 
during  the  exercise? 

•  Did  anyone  in  your  group  hoard  information?  If  so, 
why?  What  was  the  effect? 

•  Did  anyone  in  your  group  spam  the  group?  If  so 
why?  What  was  the  effect? 

•  What  types  of  team  member  behavior  are  not 
effective  in  particular  organization  structures? 

•  How  should  an  ideal  team  member  behave  in 
particular  organizational  structures? 
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Participating  Research  Institutions 


Civilian 

•  Boston  University 

•  Harvard  University 

•  Loyalist  College,  Canada 

•  University  of  South  Hampton,  UK 

•  York  University,  Canada 
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Participating  Research  Institutions 


Military 

•  Army  Research  Labs  (ARL) 

•  Cranfield  University 

•  Defense  Research  and  Development ,  Canada 

•  Defense  Academy  of  the  United  Kingdom 

•  George  Mason  University 

•  Johns  Hopkins  University 

•  Military  Polytechnic  Academy,  Army  of  Chile 

•  National  Defense  University 

•  Naval  Post  Graduate  School 

•  Portuguese  Military  Academy 

•  Singapore  Armed  Forces  Centre  For  Military 
Experimentation 

•  US  Army  War  College 

•  US  Military  Academy 


Next  Steps 


•  Continuing  feedback  loop  -  bringing  new  research 
to  classroom 

•  Teaching  the  new  skill  of  how  to  optimize  the 
organizational  approach  for  the  task  and  situation 

•  Individualized  training 


ELICIT 


V2.5 
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For  More  Information 


The  CCRP  makes  ELICIT  available  free  of  charge  to  all 
members  of  the  ELICIT  Community  of  Interest  (COI) 

Information  about  joining  the  COI  is  available  at 

http://www.dodccrp.org/html4/elicit.html 


For  more  information  about  the  CCRP  or  ELICIT 
contact  Dr.  David.Alberts  [at]  osd.mil  or 
Mary  [at]  azigo.com 
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